Skip to content

feat(c): export a pinned table's Arrow schema - #1331

Merged
JingsongLi merged 1 commit into
apache:mainfrom
suxiaogang223:codex/export-table-arrow-schema
Nov 5, 2026
Merged

JingsongLi merged 1 commit into
apache:mainfrom
suxiaogang223:codex/export-table-arrow-schema

Conversation

@suxiaogang223

Copy link
Copy Markdown
Member

Purpose

Linked issue: Closes #1328.

Native callers need to construct Arrow batches matching the pinned table.
Exporting its write schema avoids duplicating the Paimon-to-Arrow mapping or
querying a catalog again.

Prepared with AI assistance.

Brief change log

Add paimon_table_export_arrow_schema in the table bindings, reusing
build_target_arrow_schema. Write an independently owned Arrow C Data
Interface schema into empty caller-provided storage. Reject null handles,
null outputs and outputs with a live release callback without replacing them.

Tests

Validated with the existing Rust 1.97.0 toolchain on macOS arm64:

  • cargo +1.97.0 test --locked -p paimon-c --lib: 95 passed.
  • Verify names, order, nullability, field IDs and representative scalar types;
    export without catalog/schema files; free the table before inspecting output;
    reject live output reuse; release and reuse storage; check null pointers.
  • The existing write/commit/read roundtrip now builds its input from the
    exported schema.
  • cargo +1.97.0 fmt --all -- --check and git diff --check: passed.
  • cargo +1.97.0 clippy --locked -p paimon-c --all-targets -- -D warnings: passed.

Not run locally: CI-pinned Rust 1.98.0 checks, the full workspace/Vortex
suite, or the Linux/MSRV matrix. The results above cover the listed commands.

API and Format

One additive C function; no core type-mapping or storage format change. The
caller supplies initialized empty ArrowSchema storage and releases the result
through its callback. The output can outlive the borrowed table handle.

Documentation

Add schema export, batch-construction guidance and release ownership to the C
integration guide. The core mapping remains authoritative, including
unsupported types.

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 088ca65 into apache:main Nov 5, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Export the pinned table's write Arrow schema through the C API

2 participants